Experimentation as a way of life: Okapi at TREC

The Okapi system has been used in a series of experiments on the TREC collections, investigating probabilistic models, relevance feedback and query expansion, and interaction issues. The TREC-6 ad hoc task was used to test an application of a new relevance weighting formula, which takes account of documents judged nonrelevant. The application was to a form of blind feedback (using the top-ranked documents from an initial search to improve the query formulation for a subsequent search, without actual relevance feedback, on the assumption that these top-ranked documents are likely to be relevant). In the routing task, the problem is one of query optimization based on a training set with known relevant documents; investigations for TREC-6 included using a form of simulated annealing for this purpose. A signi®cant feature of this work is the need to avoid over®tting of the training sample. In the interactive track, methodology remains the major problem: we do not yet know how to conduct controlled laboratory experiments which provide good information about information retrieval interaction. The Okapi team has been particularly interested in the relation between the functionalities associated with relevance feedback and the ability of searchers to make use of these functionalities. TREC provides an excellent environment and set of tools for investigating automatic systems; its value for interactive systems is not yet proven. # 1999 Elsevier Science Ltd. All rights reserved. Information Processing and Management 36 (2000) 95±108 0306-4573/99/$ see front matter # 1999 Elsevier Science Ltd. All rights reserved. PII: S0306-4573(99)00046-1 www.elsevier.com/locate/infoproman * Corresponding author. Present address: Microsoft Resarch Ltd, St George House, 1 Guildhall Street, Cambridge CB2 3NH, UK. Tel.: +44-1223-744-769; fax: +44-1223-744-777. E-mail address: [email protected] (S.E. Robertson). 1 Present address: Microsoft Research Ltd, St George House, 1 Guildhall Street, Cambridge CB2 3NH, UK. 2 Present address: University of Sheeld, UK.